Assessing Software Cost Estimation Models: criteria for accuracy, consistency and regression

One of the problems in software cost estimation is how to evaluate estimation models. Estimation models are usually evaluated against two attributes: estimation accuracy and estimation consistency. A number of measures are reported in the literature, but they have shortcomings. There is no generally accepted standard to evaluate estimation models and the existing measures sometimes are not consistent among themselves. This paper examines existing measures of estimation accuracy and consistency and proposes two new ones: the weighted mean of quartiles of relative errors (WMQ) as a measure of accuracy and the standard deviation of the ratios of the estimate to actual observation (SDR) as a measure of consistency. Besides, a new regression criterion is proposed to determine model parameters. The proposed measures and criterion were tested with a data set from real world software projects. Results obtained show that these new measures and criterion overcome many of the difficulties of the existing ones. INTRODUCTION Since the early 1950s, software development practitioners and researchers have been trying to develop methods to estimate software costs and schedules (Abdel-Hamid, 1990). Software cost estimation models have appeared in the literature over the past two decades (Wrigley et al, 1991). However, the field of software cost estimation is still in its infancy (Kitchenham et al, 1990). The existing cost estimation methods are far from standardised and reliable (Rowlands, 1989). There is a need to evaluate estimation models and improve modeling processes. This paper will focus on how to quantitatively evaluate software cost estimation models. A new approach to determine model parameters is also proposed. In the field of software cost estimation, estimation models are usually evaluated against two attributes: estimation accuracy and estimation consistency. The rules or measures needed to describe these two attributes will be discussed in this paper. Measurement may be used for two purposes: assessment and prediction (Fenton, 1994). For assessment, the results of comparing different models may be used to judge which theory is more successful at explaining the behaviour of cost factors (Pfleeger, 1991). Successful models would provide further insight into software development processes. To assess models, we need to have common standards. Although some measures for estimation accuracy and consistency have been introduced in the literature, they are not generally accepted for model assessment (Verner et al, 1992). Sometimes these measures are not consistent among themselves. For example, a model may be better than another one with respect to one measure but poorer with respect to a different measure. For prediction, the measurement may provide feedback to improve the modeling process so that the model can satisfy the rules of measurement as far as possible. Modeling is associated with measurement (Kan et al, 1994). In this context, it is necessary to define the procedures for determining model parameters and interpreting the results (Fenton, 1994). This leads to the problem of how to determine the model parameters. Unfortunately there is also no generally accepted criterion for researchers to follow in the modelling process. This paper, therefore, will also discuss the criteria for determining model parameters. The paper is organised into three parts. The first part examines the measures used to evaluate estimation models. To overcome the shortcomings of existing practice, two new measures are proposed: the weighted mean of quartiles of relative errors (WMQ) which provides a measurement of accuracy and the standard deviation of the ratios of the estimate to actual value (SDR) which provides a measurement of consistency. The second part examines traditional mathematical procedures for formulating costing models and proposes a new regression criterion called least sum of logarithmic ratios of estimate to actual value. This is an unbiased method of finding parameters in a cost estimation model when the functional form of the model is known. The third part assesses the proposed measures and criterion. A data set from real world software projects was used to examine the proposed measures and the regression criterion so as to demonstrate their advantages over the other measures and criteria currently reported in the literature. AJIS vol S no 1 September 1997